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SWMABY 


A fundamental  question  about  instructors  Euid  Instructional  techniques 
Is  vbether  there  are  enoxigh  differences  between  Instructors  to  require  our 
concern.  It  Is  conceivable  that  such  differences  are  quite  minimal  and 
have  little  effect  due  to  the  similar  training  and  backgrounds  of  the 
instructors.  Identical  syllabi, and  standEirdizatlon  procedures.  If  such  Is 
the  case,  elaborate  instructor  tralnir.g,  selection,  or  monitorlxig  Is  un- 
necessary. If  there  are  considerable  differences  among  instructors  In 
instructlorx^L  proficiency,  action  is  required  for  efficiency  In  training. 
This  study  concerns  Itself  with  the  question  of  the  extent  of  differences 
between  instructors. 

Because  stud.en?.5  a''  r.ot  likely  to  be  completely  adequate  Judges  of 
Instructors  and  super sor 3 sure  not  likely  to  have  sufficient  opportunities 
for  observation  ot  inatruc-cion,  a measure  for  instructor  differences  was 
developed  and  used.  We  have  Judged  Instructor  differences  on  the  basie  of 
how  well  an  Indlvidtial  instznictor' s students  conq>are  with  the  sx^entr.  of 
other  instructors.  The  cocqiarlson  was  made  on  what  should  be  the  critical 
test  of  a flight  Instructor  — hov  well  his  students  performed  cub  flyers. 

The  flight  performance  of  the  students  that  12  officers  had  instructed 
through  the  first  11  flights  was  coinpared  on  the  grades  cusslgned  the 
students  on  check  flights  (their  12th  through  19th  flight)  and  their  second 
stage  of  training. 

It  was  fo\ind  that  the  performance  of  students  of  different  instructors 
was  significantly  different.  This  could  not  be  attributed  to  differences 
between  students  uslgned  to  the  instructors. 

Our  data  Izsdlcate  that  sons  ^^structors  are  not  doing  as  effective  a 
Job  of  teaching  students  to  fly  as  others.  It  follows  that  sous  cadets 
are  being  unfairly  penalized  by  this  fact. 

Such  Instzwtar  differences  can  be  minimized  by  Instructor  selection, 
instructor  training,  standardization  and  monitoring  of  procedures,  or 
instructor  ellalnation.  It  is  quite  likely  no  one  of  these  procedures  will 
be  successful  by  itself,  but  efforts  along  sdl  of  these  avenues  can  ef- 
fectively eliminate  these  differences. 


TECHNICAL 
SUPPORTI 
TATA 


lOTJRODUCnON 


Perhaps  one  of  the  more  difficult  problems  in  research  dealing  with 
the  evaliiation  of  instructor  effectiveness  is  showing  or  "proving"  a 
significant  difference  between  st'odents  of  various  instructors  as  a zesult 
of  received  instruction.  Kearly  all  the  classically  used  measures  are 
suspect.  For  exai^ple,  student  ratings  may  be  questioned  on  the  grouxids  of 
the  possible  Incosgpetence  of  the  student  to  Judge  dite  to  lack  of  e:q>erleace 
with  the  subject  matter  or  bias  on  such  "irrelevancies"  as  the  personsJ. 
appearance  of  the  iiistiructor.  Frequently,  supeirvlsor  ratings  can  be 
questioned  on  the  ground  of  limited  observation  opportunities  of  the  siqper^ 
visor.  Although  instructor  self-evaluations  have  been  used,  they  lack  a 
needed  objectivity. 

There  seems  to  be  one  alternative  tecimlqub  which  may  resolve  this 
dil'enma.  This  would  be  an  evaluation  of  instructional  proficiency  through 
student  achie'.*emsnt.  Such  a measure  would  have  a nuniber  of  advauxtages, 
the  most  Ispcrtant  being  its  validity.  After  all,  the  most  Isportant 
result  of  te/ichlng  is  not  how  well  the  Instructor  is  "liked"  by  Ms  students, 
his  superiors^  or  hlmcelf.  Of  more  isiportance  is  the  effect  his  teaching 
has  on  the  subsequent  jterformance  of  the  learner. 

In  the  final  analysis,  the  most  laportant  criterion  of  instnictor 
effectiveness  is  student  achievement  — » including  attitudes  and  under- 
standing as  well  as  skills.  Although  stated  in  a variety  of  different  ways, 
for  many  years,  professional  educators  have  agreed  upon  the  criterion  of 
student  achievement  as  a ueasur'^i  of  Instructor  successfulness.  The  recent 
report  of  Americar  Sdxicatioual  Research  Association's  Committee  on  Criteria 
of  IDeacher  Effectlr?ness  indicated  the  criterion  of  teachers’  effect  on  pupil 
achievement  was  one  of  the  more  ultimate  criteria  of  the  "ultimate-proximate" 
continuum  cf  Educational  Research.  22:  258-265,  June,  1951)* 

It  is  probable  that  the  failure  to  tise  student  performance  as  a 
mec^ure  of  instructional  proficiency  results  from  difficulties  of  meaisxire- 
ment  rather  than  the  logic  of  the  technlqua.  For  example,  a final  esc^-  ^ 
nation  is  constructed  by  an  Instructor  or  supervisor  and  administered  to  a* 
class  which  is  below  average  in  aptitude  or  original  Interusts.  Scores  are 
then  compared  with  an  examination  administered  to  stidents  of  a different 
instructor.  Such  an  analysis  would  not  be  particularly  revaallng.  Our 
reasoning  in  the  above  example  would  be  cub  foUcvs:  if  tbs  dlffsrances 

among  the  students  of  the  inetructors  were  not  controlled,  Ixtdividual 
student  differences  would  be  measured  rather  than  Instructional  dlffezences. 
Further,  if  the  instructor  or  the  supervisor  constx'ucted  or  administered 
the  final  examination,  this  measure  would  probably  reflect  what  the  In- 
structor or  the  supervisor  thought  should  be  taught  rather  than  what  the 
student  actually  learned. 

The  best  type  of  measure  would  necessarily  be  one  which  objectively 
measured  perfonmnce  directly  related  to  the  task  to  be  learned,  and  wus 
independent  of  unique  student.  Instructor,  or  supervisor  differences  con- 
csming  what  had  been  or  should  have  been  learned.  These  conditions  must 
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'be  net  if  tbese  measiires  are  to  'be  used  to  compare  differexiees  among  tbe 
instructors  in  tbe  ability  to  teach  students  in  a common  task. 

Recently,  the  investigators  had  cm  opportxmlty  to  obtain  measures  of 
student  perfocrmanee  which  seemed  to  meet  most  of  the  criteria  required  to 
eiqploy  this  measure  as  a measure  of  inst:ruetional  differences.  This  paper 
is  a report  of  the  use  of  this  type  of  measure. 

The  jnirpose  of  this  report  is  to  describe  a test  of  the  null  hypothesis 
concerning  instructor  differences  as  they  affect  student  performance  in  a 
eonq>lex  learning  situation.  Stated  in  another  fashion:  there  is  to  differ- 

ence 'between  instxnietors  in  their  effect  on  student  learnlsg  in  a ccasplex 
learning  situation.  If  differences  do  exist,  we  would  'be  in  e position  of 
saying  that  instructional  differences  play  a role  in  the  efficiency  of 
learning.  If  the  null  hypothesis  is  found  tenable,  our  attention  must  be 
directed  then  to  posslhlllties  that:  (a)  instructor  differences  in  the 

particv'^jur  situation  were  minimal;  (b)  student  learning  in  this  particular 
sit\iatlon  was  essentially  independent  of  instructor  technique;  (c)  in- 
structors so  effectively  distribute  their  training  that  instructor  differ- 
ences were  compensated  for;  or  (d)  perhaps  the  measure  was  not  sensitive 
enouc^  to  reflect  differences  in  student  perfomausce. 


fROCKDORS 


This  study  was  carried  out  in  the  Baval  Air  Training  Command.  The 
complex  task  to  be  learned  was  that  of  flying  an  aircraft.  Student  aubjeets 
were  Haval  Aviation  Cadets;  the  instructors  were  Kaval  officers  assigned  to 
the  Command  for  instructional  duty.  Although  the  instructors  had  extensive 
training  and  eiqperlence  in  flying,  as  a group  they  had  very  limited  edu- 
cation in  msthods  of  instruction. 

The  period  covered  by  this  investigation  was  between  approximately 
?cbz"aary  19^2  and  Juto  19?? Duriz^  this  period,  tbe  records  of  the 
students  trained  at  one  of  the  Initial  training  fields  were  sorted  on  the 
basis  of  their  instructor.  Than,  the  students  who  had  had  the  sems'  in- 
structor through  the  first  11  flicks  were  selected.  Tbe  final  data  for 
this  stz^  were  the  student  records  of  12  irotructors  who  had  four  or  more 
students  dinrlng  the  period  considered. 

The  students  were  given  instructions  for  11  flights  by  their  instruc- 
tors. This  is  a phase  of  training  during  which  the  student  is  grounded  on 
the  basic  fundamentals  of  flight  {landings,  take-c>ffs,  turns,  taxiing,  etc.). 
At  the  end  of  this  period,  he  is  than  given  a '•‘check  flight"  by  another  in- 
structor. This  is  designated  as  A-I2  flight.  After  successful  completion, 
the  student  receives  six  more  dual  fjlig^ts,  usuaLLly  accompanied  by  different 
instructors.  He  is  then  given  a furithef  independent  "check  flight"  by 
an  instructor  other  than  the  one  he  has ‘ previously  had.  This  is  the  A-19 
fliglit.  After  hi.s  solo,  he  then  receives  one  or  two  nvore  flights  and  pro- 
ceeds to  his  "B  stage"  of  training.  This  aac’ur.t3  to  some  1?  flights  in 
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vhlch  ha  "practlcea"  previously  learned  techniques.  This  phase  Is  known  as 
the  precision  flying  training.  In  addition  to  "practicing”  the  funda- 
aentals  he  has  learned^  he  also  learns  to  recover  from  stalls,  to  Tnake 
small  field  lemdlngs,  asA  so  on. 

The  progress  described  above  Is  somewhat  idealized  In  the  sense  that 
'‘extra  time"  is  allotted  individuals  who  are  not  proficient  in  various 
techniques  at  various  stages.  For  example.  If  the  man  Is  not  considered 
safe  to  solo  on  the  A-19  flight,  and  he  Is  considered  potentially  capable, 
he  may  be  given  several  extra  flights,  rc -checked,  and  permitted  to  solo. 

A man,  ho:^ver,  can  be  dropped  if  found  considerably  wanting  at  any  given 
stage. 


Tbs  grades  of  toe  A-12  flight,  the  A-I9,  and  the  grades  made  during 
the  "B  Stage"  of  training  were  used  as  criteria.  As  Indicated,  the  grades 
given  during  the  A-12  and  the  A-I9  flights  are  essentially  progress  checks 
to  evaluate  student  learning.  In  x sense,  tbc  ^3  Stage"  Is  a practice 
period  and  an  extension  of  learning  during  the  early  stage  of  trclning.  As 
was  Indicated,  these  grades  are  assigned  independently  of  the  orlglual 
Instructor.  They  should  serve  as  an  evalxiation  of  the  proficiency  of  the 
individual.  Independent  of  tue  instructor. 

The  grades  are  averages  of  the  ratings  by  instructors  on  various 
maneuvers  on  which  the  student  has  had  training.  If  the  performance  of  a 
particular  phase  (e.g., landings)  is  unsatisfactory,  then  the  student  la 
given  a ^:rade  of  1.0  on  that  maneuver.  If  below  average,  a grade  of  2.0 
is  If  average,  3*0,  and  If  above  average,  the  grade  of  4^.0  Is 

signed  that  maneuver.  The  grade  for  a given  flight  Is  the  svot  at  these 

separate  ratings  divided  by  the  nuiA>er  of  maneuvers  which  were  rated.  Zbs 
average  of  the  grades  Is  approxtnstely  3>0,  and  a grade  of  2.5  for  a given 
flight  is  considered  unsatisfactory  and  the  student  is  either  dropped  tram 
training  or  given  extra  time.  These  grades  should  reflect  how  well  the 
Individual  has  learned  to  perform  the  task  for  which  he  has  been  Instructed;, 
namely,  efficiently  flying  an  aircraft. 

In  several  ccMes,  a student  or  an  instructor  attrited  before  the  ‘*B 
Stage."  In  these  cases, he  was  assigned  a grade  of  2.3  (considered  unsatis- 
factory), and  Included  In  the  determination  of  the  student  record  of  the 
instructor. 

An  analysis  of  variance  was  performed  on  these  grades  by  Instructors. 

A second  analysis  of  variance  of  the  aptitude  scores  cf  the  students  of 
each  Instructor  was  performed  to  check  the  assumption  that  no  Initial 
differences  existed  among  the  students  of  the  various  instructors. 

A brief  summary  of  the  preceding  description  of  our  procedure  le  as 
follows : 

a.  Instructors  - 12  experienced  Haval  Aviators. 

b.  Students  - 4,  3,  or  6 traiiises,  without  previous  flight 
experience,  assigned  to  each  Instructor. 
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c«  Instruction  - 11  training;  flights  of  apprcxisoately 
ons  hour  and  twenty-five  minutes  each  in  which  the 
instructor  attempts  to  give  each  student  the  basic 
fundamentals  of  flying  an  aircraft. 

d.  Measures  - Evaluations  of  student  ability  to  fly  h 
Naval  aircraft  immediately  after  the  11  initial 
flights  (the  A-12  flight);  a further  progress-  check 
of  the  student's  flight  proficiency  after  l8  flights 
(the  A-I9  flight);  the  average  performance  in  a 
second  stage  of  training  in  which  the  student  learns 
to  become  more  precise  and  de-tailed  in  his  flying 
techniques  (B  Stage).  This  latter  measure  consists 
of  some  21  flights  which  the  student  performs  after 
having  bad  2k  previous  training  missions.  Each  of 
the  meesvires  used  were  given  by  Instructors  o-cher 
than  the  original  instructor. 

e.  Controls  - It  was  assumed  that  no  individual  differ- 
ences existed  between  the  stuients  of  the  various 
instructors.  Tills  assumption  was  tes-ted  suid  reported 
in  our  results.  Further;  it  is  reasonable  that  the 
measures  taken  are  iniepenient  of  any  unique  Ins-tructor 
or  supervisor  bias  in  regaz*d  to  what  the  student  should 
have  learned  or  had  learned,  since  the  graidlng  system 
has  been  developed  independent  of  the  instructors  and 
their  Imaedlate  siqiervlsors.  Finally,  it  can  be  easily 
maintained  that  the  measures  were  rela-ted  to  the  task 
to  be  learnsd. 

RESULTS 

Ihs  means  of  the  student  flight  grades  on  the  two  cheek  flights  (A-12 
and  A-19)  and  the  latter  stage  of  training  ("B  Stage")  for  the  12  in- 
structors are  given  in  Tsble  1.  Further,  the  means  of  the  Flight  Aptitude 
Bating  ncores,  a msasure  of  flight  aptitude,  are  given  in  Tsble  I.  The 
table  also  shows  the  nuidber  of  students  on  which  each  measure  vas  based. 

An  F wiMt  test  of  tbs  homogeneity  of  variance  for  tl^  three  check  hops 
was  completed,  l^re  was  no  significant  difference  in  the  -variance  of  the 
three  grotps,  l.e.,  it  may  be  concluded  that  the  variances  are  homogeneous. 
A problt  analysis  of  the  three  sets  of  ratings  was  performed  and  indicated 
that  the  distributions  of  check  ride  scores  are  approximately  normal.  An 
analysis  of  variance  between  the  students'  Fll^t  Aptltiide  Rating*  scores 
was  completed,  and  is  given  in  Table  II.  There  was  no  statistically 
significant  difference  between  students.  The  FAR  is  predictive  of  success 
In  the  Flight  I^ednlng  Program,  and  this  information  can  be  interpre-ted  to 


* Bereafter  referred  to  as  FAR.  It  is  a composite  score  bsised  on  three 
different  measures  purpoirtlng  to  predict  success  in  the  FT  progrimi. 
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mean  that^regaurdlng  the  predicted  ability  to  conplete  the  progrsn  success- 
fully, the  etiidents  of  one  Instructor  did  not  differ  frt»n  the  stxidents  of 
another  Instructor.  Tbsretoee,  any  differences  vhich  are  found  in  final 
achleveisazit  must  he  esq^lalned  as  resulting  from  some  effect  other  than 
shinty  to  coiqplste  the  Flight  Training  Progrss^  The  F max  test  of  hoax>- 
geneity  of  varlazxce,  the  problt  analysis,  and  the  analysis  of  variance  of 
FAR  scores  represent  tests  of  assumptions  underlying  the  anal.ysls  of  vari- 
ance technique.  On  the  beisls  of  thle  Information,  It  vas  concluded  that 
further  analysis  of  data  would  be  Justified. 

Accordingly,  an  analysis  of  variance  ms  performed  hetw'ien  the  A-12, 
A-18,  and  B stage  check  ride  grades  for  62  students  who  bad  been  assigned 
to  12  different  Instructors.  Because  of  the  unequal  frequencies  in  the 
cells,  Tsao'e  method  of  solution  was  utilized  for  the  result  In,;  4 x 12 
design  (Psychometrlka.  11:107-128,  June,  1946). 

The  restilte  of  this  analysis  are  shown  in  Table  III.  The  interaction 
term  (student  achievement  x instructors)  was  not  significant.  Therefore, 
It  was  pooled  with  the  "within  variance”  for  testing  the  signlflcemce  of 
the  main  effects  (variation  between  instructors,  and  variation  between 
student  achievement).  Differences  significant  at  the  .01  level  were  found 
between  instructors  and  between  student  achievement  on  the  three  check 
rides.  The  results  of  this  study  indicate  that  there  Is  a difference  be- 
tween instructors  regarding  the  achievement  of  their  students,  and  there 
Is  a difference  between  stxidente  in  achievement  from  check  ride  to  check 
ride. 


C0HCLU8I0K5  AHD  DISCU3SI0H 

It  will  he  recalled  that  the  null  hypotheele  tested  In  this  Investi- 
gation vas  coneemsd  with  Instructor  differences  as  they  affect  student 
performanee.  Pleasures  were  obtained  of  student  performance  which  were 
Independent  of  the  evaluation  of  the  student  by  the  instructor  or  the 
Instructor  by  the  student.  It  was  found  that  significant  differences 
existed  between  the  perfomanees  of  different  Instructors,  euad  the  null 
hypothesis  was  rejected.  It  Is  clear  that  the  differences  between  in- 
structors significantly  affected  performance  of  students. 

Our  finding  nay  be  little  more  than  a proof  of  something  we  all  agree 
on.  We  hope,  however,  that  these  data  will  serve  as  a twofold  reminder: 

(1)  that  Instructor  differences  do  result  in  student  achievement  differences, 
and  (2)  that  methodologically  these  differences  can  he  established  inde- 
pendent of  the  arguments  surrounding  the  other  techniques  of  evaluating 
instructor  effectiveness* 

!lbe  example  of  complex  learning  (aircraft  flying)  night  be  considered 
a special  case  in  that  tue  instructor -student  relationship  was  on  a face  to 
face  relationship,  rather  than  in  a claseroom  group  situation.  However, 
the  flight  student  has  bad  no  previous  experience  or  training  .In  flying 
alrcredTt.  It  is  one  of  the  few  learning  situations  which  is  available  which 
lends  itself  to  the  evaluation  of  the  effects  of  one  inatznictor.  Such  a 
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situation  has  some  Import  In  other  educational  situations.  Basically,  the 
teaching  of  the  haslc  skills  entails  a teacher-student  relationship  vhich  is 
quite  independent  of  group  effects.  One  has  but  to  look  at  the  vealth  of 
educational  literatvure'vto  see  the  importance  attached  to  the  "individual- 
ization of  instr'jcti'fep^i^  "teaching  for  individual  differences,"  and  "pro- 
motirg  ind-ytlual  atterirlon."  However,  the  teaching  usually  referred  to 
docb  vtjjie  place  in  a group  sit’iation,  and  it  is  difficult  to  ascertain 
accurately  the  effects  ins true tor . Further,  after  an  Initial  learn- 

ing experience  Ir  or.  ox  che  basic  skill  subjects,  the  effects  of  later 
instruction  are  impossible  to  ascertain.  In  o^ir  opinion,  this  particular 
study  offers  definitive  evidence  of  the  effects  of  Instructors,  and  gives 
objective  evidence  to  the  contention  that  different  instructors  do  have  a 
differential  effect  on  istudant  learning.  Tliis  principle,  it  is  felt,  can 
be  generalized  to  learning  situations  in  which  basic  skills  are  taught. 

As  such,  this  study  represents  evidence  regeurding  the  Importance  of  the 
instructor,  a concern  for  instructor  methods,  and  thereby  a concern  fox' 
instructor  education. 
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FLIGHT  APTITUr®  RATINGS  AND  GRADES  FOR  THE  A-12,  A -19,  AND  B STAGE  FLIGHTS 
FOR  62  STUDENTS  ASSIGNED  TO  12  INSTRUCTORS 


Instructor 


TABLE  II 

ANALYSIS  OF  VARIANCE  OF  FLIGHT  APTITUDE  RATINGS 
OF  62  STUDENTS  ASSIGNED  TO  12  INSTRUCT(»S 


Source 


df 


Svun  of  squares 


Mean  souare 


Between  tnatnictors 


11 


30-6178 


2.78 


< 1 


Wlthir:  groups 


50 


144.8822 


>.90 


Total 


61 


TABJ^  III 


ANALYSIS  OF  VAKIABCE  CP  62  STUDEKTS  ASSIGNED  TO  12  INSOSUCTOPiS 
ON  IBREE  SETS  CF  CHECK  FLIGHT  (HIADES:  A-12,  A-19  AND  B STAGE 


So\irce 

df 

Svim  of  Squares 

Mean  Sqviare 

F 

Between  flights 

2 

1.957059 

.9785 

18.55**" 

Between  Instructors 

11 

5.061^799 

.2786 

5.28?^* 

Peald\xal 

172 

9.074092 

.05^8 

Interaction 

22 

.361892 

.0592 

< 1 

Error 

150 

8.2122 

.0547 

Total 

185 

Significant  at  the  .01  level. 
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NOTICE:  WHEN  GOVERNMENT  OR  OTHER  DRAWINGS.  SPECIFICATIONS  OR  OTHE 
ARE  USED  FOR  ^kNY  PURPOSE  OTHER  THAN  IN  CONNECTION  WITH  A DEFINITELY 
GOVERNMENT  PROCUREMENT  OPERATION,  THE  U.  S.  GOVERNMENT  THEREBY  H‘ 
NO  RESPONSIBILETY,  NOR  ANY  OBLIGATION  WHATSOEVER;  AND  THE  FACT  THAT 
GOVERNMENT  MAY  HAVE  FORMULATED,  FURNISHED  OR  IN  ANY  WAV  SUPPLIED 
SAID  DRAWINGS,  SPECIFIC.^TIONS,  OR  OTHER  DATA  IS  NOT  TO  BE  REGARDED  BY 
IMPLICATION  OR  OTHERWISE  AS  IN  ANY  MANNER  LICENSING  THE  HOLDER  OR  At. 
PERSON  OR  CORPORATION,  OR  CONVEYING  ANY  RIGHTS  OR  PERMISSION  TO  MANI 
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